Skip to content

feat(wasm): add WebAssembly build and @columnar-tech/dbc-wasm npm package - #395

Merged
zeroshade merged 37 commits into
mainfrom
wasm-js-interface
Jun 25, 2026
Merged

feat(wasm): add WebAssembly build and @columnar-tech/dbc-wasm npm package#395
zeroshade merged 37 commits into
mainfrom
wasm-js-interface

Conversation

@zeroshade

Copy link
Copy Markdown
Member

Summary

Adds a WebAssembly (GOOS=js GOARCH=wasm) build of dbc plus a Node-importable npm package, @columnar-tech/dbc-wasm, so applications can search registries, resolve versions, install/uninstall ADBC drivers to disk, and verify signatures in-process — without spawning the CLI subprocess.

What's included

  • Go wasm build (wasm/): fetch-backed HTTP transport (roundtripper_js.go), a Promise bridge (bridge_js.go), the JS API surface (api_js.go, ops_node_js.go), and Node (-tags dbcnode) + browser entrypoints. http.DefaultClient/dbc.DefaultClient are routed through the fetch transport so auth-internal OAuth refresh works under wasm.
  • npm package (packages/npm-wasm/): CJS/ESM/TS loaders, a shared Go/wasm bootstrap (boot.cjs), and one canonical client surface (search/resolve/install/uninstall/listInstalled/verifySignature/close).
  • Per-instance client handles: baseURL/credential are isolated between loadDbc() instances and refreshed tokens persist across calls.
  • Optional worker backend (loadDbc({ worker: true })) running the runtime in a worker_threads Worker, with a robust lifecycle: idempotent close(), init-failure teardown (no leaked worker), and RPC rejection on close/exit instead of hanging.
  • Windows-host path groundwork (normalizeLocation, splitConfigList seam) — experimental.
  • CI (.github/workflows/wasm.yml): cross-OS (ubuntu/macos/windows) build + smoke + worker-smoke + unit tests.

Phasing

Phase 0 (de-risk spike — see wasm/spike/REPORT.md) → Phase 3 (npm package) → Phase 4 (response streaming, Windows groundwork, worker backend). Review findings from each phase were addressed in focused follow-up commits.

Testing

  • node packages/npm-wasm/test/{smoke,worker.test,normalize.test}.cjs — all pass (in-process, worker, and path-normalization).
  • Both wasm build tags compile (GOOS=js GOARCH=wasm with and without -tags dbcnode).
  • Native build/tests unaffected.

Known limitations / follow-ups

  • Browser build is a compiled seam but not yet delivered (no loader, npm entry, or smoke test); tracked as a future phase. packages/npm-wasm/index.d.ts is the single canonical type declaration.
  • Windows host support is experimental; the windows-latest CI job exercises the drive-root os.MkdirAll/os.OpenRoot round-trip as a non-continue-on-error signal.
  • install/listInstalled use the process-global platform (no per-call override); verifySignature trust is anchored to SignedByColumnar by design.

@zeroshade zeroshade changed the title feat(wasm): WebAssembly build + @columnar-tech/dbc-wasm npm package feat(wasm): add WebAssembly build and @columnar-tech/dbc-wasm npm package Jun 19, 2026

@amoeba amoeba left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I did a quick scan and found one issue related to credential paths.

As an aside, which version of dbc will we recommend people use for Node.js?

Comment thread wasm/api_js.go
Comment thread .github/workflows/wasm.yml Outdated
Comment thread .github/workflows/wasm.yml Outdated
Comment thread .github/workflows/wasm.yml Outdated
Comment thread .github/workflows/wasm.yml Outdated
Comment thread .github/workflows/wasm.yml Outdated
Comment thread .github/workflows/wasm.yml Outdated
Comment thread config/splitlist_js.go

package config

// splitConfigList does not split under GOOS=js: the wasm API passes a single

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really needed? Since SplitList only ever runs on a single platform, mixing isn't an issue? Not sure if a change is needed here but just commenting.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is, but not because of platform mixing — it's what GOOS=js does to filepath. Under js, filepath.ListSeparator is : (js is treated as a Unix target regardless of the actual host OS), so on a Windows host a drive-lettered location like C:/drivers would split into ["C", "/drivers"]. The wasm API always passes a single explicit directory, so the correct behavior under js is "don't split," which is what this override does; native builds keep the normal filepath.SplitList in splitlist_other.go.

Comment thread packages/npm-wasm/scripts/build.js Outdated
zeroshade added 10 commits June 24, 2026 16:08
Compile dbc to GOOS=js GOARCH=wasm and expose a Node-importable library
covering search, install-to-disk, list, uninstall, and signature
verification, plus private-registry authentication.

- telemetry: build-tagged telemetryMachineID() returns "" under js
  (machine-id has no wasm port); removed its direct import from
  client.go and drivers.go
- client: export WithCredentialResolver for host-injected credentials
- config: SetPlatformTupleOverride (js-only) sets the host platform
  tuple, since GOARCH=wasm otherwise yields unknown_wasm64
- wasm/: fetch-backed http.RoundTripper, a Promise+goroutine bridge, and
  the Node entrypoint (search/install/list/uninstall/verify). The
  entrypoint routes http.DefaultClient through fetch so dbc's auth
  refresh, license, and OIDC calls work under Node
- wasm/spike: Node harnesses, report, and run script validating the
  gate end-to-end (config paths, search over fetch, install/list/
  uninstall on disk, gopenpgp signature verify, 401 -> refresh -> retry,
  and no event-loop deadlock under concurrency)

Native build and dbc/config/auth tests are unaffected.
… test

Refactor the wasm entrypoint so a future browser build is a cheap addition
and shares logic with the Node build:

- api_js.go (js): shared handlers (search/resolve/verify), client setup,
  JS config hooks, and the http.DefaultClient -> fetch override
- ops_node_js.go (js && dbcnode): install/list/uninstall (filesystem)
- main_node.go / main_browser.go: thin entrypoints. The browser build omits
  ops_node_js.go entirely, so install/extract code is excluded (browser
  binary is ~338 KB smaller than the node build)
- add resolve(name, platform) returning versions + latest package URL
- add wasm/dbc.d.ts TypeScript declarations
- add native unit tests for WithCredentialResolver

Verified: native build + dbc/config tests pass; vet clean for native and
both js builds; node harness regression green.
… race

- search/resolve (wasm/api_js.go): no longer reject the Promise on a partial
  Client.Search error (one registry fails while another succeeds). Search now
  returns {drivers, warning} and rejects only when zero drivers resolve,
  mirroring the CLI --json warning contract; resolve proceeds with partial
  results so a reachable registry still answers
- list (wasm/ops_node_js.go): use the new env-free config.FindDriverConfigsIn
  so concurrent dbcList calls no longer race on the process-global
  ADBC_DRIVER_PATH
- config: add FindDriverConfigsIn(location) plus a native unit test
- update dbc.d.ts (SearchResult) and the spike harnesses for the new
  search return shape
… 6510)

jsResolve guarded the early-return on a partial Client.Search error, but when
the exact driver was absent from the partial results it fell through to a plain
"not found" that hid the registry failure. Search does substring matching, so a
reachable registry's non-exact hits could mask an outage on the registry hosting
the exact driver. Now wrap searchErr in the not-found error; return plain
"not found" only when search completed cleanly.
Node-importable packaging for the wasm build, separate from the native-binary
@columnar-tech/dbc wrapper.

- loader (index.cjs + index.mjs): wires Node fs/process/webcrypto and
  go.env=process.env before wasm_exec.js (the two Phase 0 R1 requirements),
  instantiates dbc.wasm, and exposes an async API (search/resolve/install/
  uninstall/listInstalled/verifySignature) that auto-detects the host platform
  tuple and JSON-parses results
- index.d.ts: TypeScript declarations for loadDbc plus result types
- scripts/build.js: compiles ./wasm (-tags dbcnode), copies wasm_exec.js from
  GOROOT, generates package.json at a given version, copies LICENSE; build
  artifacts are gitignored, mirroring the packages/npm conventions
- test/smoke.cjs: end-to-end smoke test through the package API against the
  repo test fixtures
- README documents usage, the Node >=18 requirement, and credential bootstrap

Verified: build + smoke pass (search/resolve/install/list/verify/uninstall);
ESM and CJS entrypoints load; npm pack yields an 8-file ~4 MB tarball.
…roborev 6519)

loadDbc previously applied baseURL/credential by mutating singleton wasm
globals, so two clients in one Node process clobbered each other: after a
second loadDbc, the first client's search/install would use the later
baseURL/credential — a cross-tenant registry/credential mixup.

- api_js.go: replace the baseURL/credResolver globals and the dbcSetBaseURL/
  dbcSetOAuthCredential setters with clientFromConfig(cfgJSON); dbcSearch and
  dbcResolve take a leading config JSON and build the dbc.Client per call
- ops_node_js.go: dbcInstall takes the config JSON; dbcUninstall uses a bare
  client (it performs no network I/O)
- index.cjs: the loader captures baseURL+credential per loadDbc instance and
  passes them to each network call; platform is set once at init as a
  process-global host constant (overridable via loadDbc({ platform }))
- spike harnesses updated for the cfg-first signatures

Verified: two clients with different baseURLs stay isolated (a.search() still
hits registry A after b=loadDbc with baseURL B); package smoke and the auth
401 -> refresh -> retry harness pass; native build/tests + vet clean.
The fetch RoundTripper buffered each response via arrayBuffer(), holding the
full driver tarball in wasm linear memory before copying it to disk. Replace
that with a ReadableStream-backed io.ReadCloser (jsStreamBody) so install copies
the tarball to disk chunk-by-chunk; ContentLength is taken from the header and
null bodies map to http.NoBody. Read awaits the stream reader, so it runs on the
existing per-call goroutine.

Verified: package smoke + spike harnesses (search/install/verify/uninstall) and
the auth 401 -> refresh -> retry harness pass; node and browser builds clean.
…orev 6521)

The roborev-6519 fix rebuilt a fresh auth.Credential from the immutable config
JSON on every call, so a 401 -> refresh updated only that throwaway credential;
the next call started from the original (stale) token again — wasteful, and
fatal with single-use refresh tokens (operations 2+ would fail).

- api_js.go: keep a per-instance dbc.Client in a handle registry. dbcNewClient(cfg)
  builds the client once and returns an int handle; search/resolve/install take
  the handle and reuse the stored client, so the credential pointer (and any
  refreshed access token) persists across calls. dbcCloseClient releases it.
- ops_node_js.go: dbcInstall takes the handle.
- index.cjs: loadDbc creates a handle, exposes close(), and registers a
  FinalizationRegistry to release handles on GC.
- index.d.ts: add Dbc.close(); spike harnesses updated (the auth harness now
  asserts a single token refresh across two searches).

Verified: tokenRefreshes==1 across two searches with a stale injected token;
two clients stay isolated (6519); package smoke + native tests + vet clean.
…ase 4)

Under GOOS=js, path/filepath uses Unix semantics on every host, so
filepath.SplitList splits on ":" and corrupts a Windows drive-lettered location
like C:/drivers. Per an Oracle architecture review, introduce a build-tagged
splitConfigList() seam (non-js: filepath.SplitList byte-for-byte; js: the single
explicit location, no split) and use it at the explicit-location sites the wasm
API traverses:

- config.EnsureLocation (install) and config.GetDriver env branch (uninstall)
- config.FindDriverConfigsIn (list)

loadConfig/getEnvConfigDir (env-var multipath, out of wasm scope) are left alone.
filepath.Join/Dir/Base/Clean/OpenRoot are already safe under js (Node fs accepts
forward-slash drive paths); filepath.IsAbs is not on the install path.

The npm loader also normalizes a Windows location at the JS boundary (backslashes
to forward slashes; drive-relative to absolute).

Native builds are byte-identical (splitConfigList aliases filepath.SplitList off
js); native + Linux-wasm tests/smoke/spike are unchanged. NOT yet verified on a
Windows runtime: os.MkdirAll/os.OpenRoot over a drive root through Node fs (see
README caveats) -- validate on a Windows runner before claiming support.
zeroshade and others added 21 commits June 24, 2026 16:08
…roborev 6551)

The in-process and worker backends each constructed their own client object
literal and had drifted: sync vs async close(), always-on vs feature-detected
install methods, and prefixed vs raw error messages. Factor both behind a
single buildClient() factory driven by a per-backend call(fn, args) dispatcher
so the public shape is defined once and cannot diverge.

- close() now returns Promise<void> and is idempotent in both backends; the
  in-process path guards against a double-free and unregisters the
  FinalizationRegistry on explicit close.
- install/uninstall/listInstalled are feature-detected symmetrically: the
  worker reports hasInstall in its ready handshake, matching the in-process
  typeof check.
- All surfaced errors are namespaced with "dbc-wasm:" via a shared
  prefixError() helper (worker RPC rejections and in-process Go errors included).
- index.d.ts/README document the remaining design realities rather than
  changing behavior: mandatory close() under worker:true (worker thread is not
  GC-managed), process-global platform vs per-instance baseURL/credential and
  resolve()'s per-call precedence, hostPlatformTuple()'s throw, normalizeLocation
  as a low-level internal export, the search-only warning rationale, a DbcClient
  alias, verifySignature(library, signature) param names, and a comment on the
  install-vs-uninstall handle asymmetry.

Verified by the normalize, worker-smoke, and in-process-smoke tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The runtime instantiation sequence (inject fs/process/crypto globals, load
wasm_exec.js, instantiate dbc.wasm, run, and verify the dbc* API registered)
was duplicated byte-for-byte between index.cjs (ensureRuntime) and worker.cjs.
Any change to the boot protocol had to land in both, and a divergence would be
a silent init bug caught only by whichever path was exercised.

Extract it into a single bootRuntime() in boot.cjs. Each caller keeps its own
error-reporting policy: the in-process path namespaces the rejection with the
dbc-wasm: prefix, the worker posts a {type:"fatal"} message. Add boot.cjs to
the published files list in the build.js package.json generator so it ships in
the tarball.

Smoke tests pass for the in-process, worker, and worker-init-failure paths.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lper

The worker backend repeated the `closed = true` toggle and pending-RPC
rejection inline across five sites (the message/fatal, error, exit handlers,
the init catch block, and close()). Extract a single idempotent
markClosed(err) helper that flips `closed`, drains `pending`, and returns
whether it won the teardown race. The event handlers and close() now call one
function instead of re-implementing the guard, removing the "did I guard
`closed` here?" class of bug (R3 / smells #3, #4).

Behavior is unchanged: the exit handler still skips readyReject when an
explicit close()/init-failure already tore down (now via markClosed's return
value), and the init catch still terminates the worker before propagating.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The init gate (`readyReject`) was rejected separately from the in-flight
RPCs (`markClosed`) in every worker failure handler, so a reader had to
hold two rejection channels in mind to confirm a failing worker can
neither hang nor double-settle. Route `readyReject` through markClosed so
every failure path settles both channels at once and each handler becomes
a one-liner. Behavior is unchanged (already-settled promises ignore later
rejects; the `closed` guard short-circuits repeats), verified by the
existing worker smoke test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(roborev 6562)

loadDbc()'s in-process path created the client via globalThis.dbcNewClient(cfg) outside the call() dispatcher, so load-time failures (e.g. an invalid credential registryURL) rejected without the dbc-wasm: namespace the worker backend and the rest of this layer apply. Wrap it in try/catch and rethrow prefixError(e) for consistent error attribution.
…orev 6563)

Adds an in-process regression assertion to smoke.cjs mirroring the worker path: loadDbc() with an invalid credential registryURL must reject with a dbc-wasm:-prefixed message, locking in the error-attribution fix from 0046b19.
- Delete stale/unreferenced wasm/dbc.d.ts; packages/npm-wasm/index.d.ts is the single canonical type declaration.
- Document main_browser.go as an experimental, not-yet-delivered future-phase seam (no loader/npm entry/test).
- REPORT.md: superseded change-set note for the main_node.go -> api_js/ops_node_js/bridge_js/roundtripper_js split; Go 1.26.4 (spike) vs CI-pinned 1.25; Phases 1-2 absorbed into Phase 3/4.
- README: reconcile the Windows caveat with the windows-latest CI smoke gate (define the signal instead of 'not verified').
- index.d.ts: document the in-process process-global platform limitation for install/listInstalled, and the SignedByColumnar-only verifySignature trust model.
…ndows

boot.cjs set go.env = process.env, which on Windows CI runners (large environments) overflowed wasm_exec.js's combined argv+env size cap, throwing 'total length of command line and environment variables exceeds limit' from go.run() before the runtime started. Forward only the vars the GOOS=js runtime needs ($HOME, $XDG_*, temp dir), mapping Windows %USERPROFILE% to $HOME. Linux/macOS smoke+worker tests unaffected.
…ows (roborev 6570)

Go built with GOOS=js resolves os.TempDir() from $TMPDIR only (falling back to /tmp), so the prior curation that forwarded %TEMP%/%TMP% under their own names left install creating a temp dir under a nonexistent /tmp on Windows. Map TMPDIR from TMPDIR||TEMP||TMP (and HOME from HOME||USERPROFILE), normalizing backslashes to forward slashes on Windows. Extract the curation into a pure curateGoEnv(env, platform) and cover it with deterministic unit tests in normalize.test.cjs.
Diagnostic for the windows-latest listInstalled failure: include the actual listInstalled() result and a recursive listing of the install directory in the assertion message so the Windows CI log shows where install output and driver discovery diverge.
…dows hosts

loadDir enumerated manifests with fs.Glob(os.DirFS(dir), "*.toml"), but under GOOS=js an os.DirFS rooted at a Windows drive path (e.g. C:/drivers) matches nothing, so listInstalled returned no drivers on Windows hosts even though the manifests were present on disk (caught by the WASM windows-latest smoke). Enumerate with os.ReadDir(dir) + a .toml suffix filter, which goes straight through the host fs and is behavior-equivalent on native. Native config tests + Linux wasm smoke/worker pass.
The WASM Windows-host path is experimental and not yet passing: driver discovery (listInstalled) returns empty under GOOS=js for drive-letter locations even after install succeeds (tracked in #396). Exclude windows-latest from the WASM workflow matrix until fixed, revert the loadDir os.ReadDir change (it did not resolve the Windows discovery issue), and reconcile the README to state Windows WASM is experimental and not CI-gated. The env-overflow and TMPDIR loader fixes remain.
Document that omitting baseURL uses Columnar's public driver CDN (https://dbc-cdn.columnar.tech) and that setting it points at a private/self-hosted registry.
…age + CI)

wasm/spike/ (REPORT.md + Node harnesses + run.sh) was a Phase 0 proof of concept. The feature now ships as @columnar-tech/dbc-wasm with a CI-gated test suite (smoke/worker/normalize), so the spike is superseded. Not referenced by CI, docs, or code; history retained in git.
…tream bug)

Bun's WebAssembly <-> node:fs callback bridge mishandles values from Go's GOOS=js runtime, so file-writing ops (install/uninstall) fail with ERR_OUT_OF_RANGE. Node >= 18 and Deno are validated; Bun is unsupported pending an upstream fix.
Add a SHA-pinned denoland/setup-deno step and run normalize/smoke/worker under 'deno run -A' after the Node steps (ubuntu + macos). Deno is a validated runtime; this gates it in CI.
Co-authored-by: Bryce Mecum <petridish@gmail.com>
The applied review suggestion left the description string unterminated
(trailing comma, no closing quote), breaking 'node build.js' with a
SyntaxError and failing the Wasm CI build on ubuntu/macos. Close the
string and keep the reviewer's intended wording.
- rename workflow to 'Wasm'
- match Go sources recursively ('**/*.go') so changes in any package
  trigger the build
- bump pinned actions to current refs: actions/checkout v6.0.2 (matches
  dev.yml) and actions/setup-go v6.5.0
go.mod requires go >= 1.26; install Go 1.26 so the wasm build works under
setup-go v6 (which pins GOTOOLCHAIN=local and so won't auto-download a
newer toolchain).
@zeroshade

Copy link
Copy Markdown
Member Author

As an aside, which version of dbc will we recommend people use for Node.js?

It's a question of use case. Similar to the question of whether they should use the dbc CLI or import github.com/columnar-tech/dbc as a dependency in a Go application:

You only really need to import the library (in this case the wasm) if you're intending to integrate dbc into your application and dynamically manage drivers at runtime but don't want to (a) bundle the dbc CLI alongside your application as a sidecar or (b) shell out to exec dbc as a separate process that you have to parse the output from. The idea is that you can just use the dbc CLI directly (in node/deno you can do npx dbc install for instance) if that works for your scenario, or you can embed the dbc library so that your published application can install a driver on the fly (at the cost of making the dbc library a dependency rather than packaging the CLI as a sidecar).

Does that make sense?

@amoeba

amoeba commented Jun 24, 2026

Copy link
Copy Markdown
Member

It makes sense but I'm still wondering why this approach versus shipping Node API bindings to github.com/columnar-tech/dbc. When I've seen this kind of thing done before it was because the browser was a target. So is this just to set dbc up for that? I'm not sure we really need two packages so maybe the current one could get deprecated.

@zeroshade

Copy link
Copy Markdown
Member Author

We might decide to change the names in the future. But this is essentially providing bindings to dbc without needing to use cgo/FFI (maybe rename the current columnar-tech/dbc to columnar-tech/dbc-cli?)

In the future we might make the package available in the browser too, but we wouldn't be able to provide any driver installation/uninstallation in the browser and it might make more sense to just provide a simple sdk to the CDN rather than making this available in the browser. The primary purpose of the wasm package is to maintain the logic for handling driver installation/uninstallation without having to reproduce it

@zeroshade
zeroshade merged commit 7d657e1 into main Jun 25, 2026
14 checks passed
@zeroshade
zeroshade deleted the wasm-js-interface branch June 25, 2026 16:21
@amoeba

amoeba commented Jun 25, 2026

Copy link
Copy Markdown
Member

Sounds good, thanks for explaining.

PS: I think we want to have whatever the primary package for dbc is be exactly @columnar-tech/dbc (not dbc-cli) and maybe even dbc if we get it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants